It seems a lot of accessibility stuff was introduced by @bollwyvl here: #294.

Nick, WDYT about the py11y removal proposed by @choldgraf here?

Btw, @choldgraf, it seems there is some branch conflict.

Yes I would love to hear @bollwyvl's thoughts re: maintainability+complexity vs. doing a reasonable job of addressing accessibility issues.

I think the main question I have is: what are the benefits to using pa11y + lighthouse, vs just lighthouse alone

i think it's a regression in development and testing robustness if there is no local capability for a developer to identify (and therefore fix) the ~30% of WCAG issues that pa11y's rules can actually say anything about. But then i haven't been doing much beyond getting that set up, as the roadmap hasn't been changed since we added it. The fact that pa11y finds those issues, and lighthouse doesn't, already suggests that it covers more than lighthouse.

To that end, I don't even know what claims lighthouse (or the wrapper action) are making, as it's more of a general-purpose tool, and was useful for putting number to things like asset pre-loading that helped us improve the perceptual speed of theme with actual metrics, and helping keep it there (provided we haven't just been bumping the thresholds down). The fact that it handles some accessibility concerns seems almost incidental, aside from google's goal of helping you convert more eyeballs (or screen readers) so they can sell more ads and personal data.

Longer term, I'd prefer that concerns like accessbility aren't delegated to a secondary test suite: i'm also looking into working accessibility testing into more real browser environments, with e.g. robotframework/selenium, but don't yet have anything concrete to demonstrate these. The advantage there is one could hijack things like Capture Page Screenshot, which get run in happy day and error situations, which would help on highly dynamic tools... which some documentation sites get closer to with e.g. thebe, jupyterlite, etc.

But same deal applies: if a tool can't be run locally and in CI worker, I feel like it's bad for open source.

Just a note that you can generate lighthouse reports directly from Google Chrome. So at least that's one option for local running, but as you say the things that lighthouse vs. pa11y test for are slightly different.

I agree with all of your points, I just worry that right now our usage of pa11y is not super actionable. As a first step, I've added some more documentation of what the pa11y error messages mean in the contributing docs: https://pydata-sphinx-theme.readthedocs.io/en/latest/contribute/topics.html#errors-in-ci-cd-and-what-to-do

Maybe this will make it easier for people to understand what the pa11y test does and how to interpret the results?

Two more updates on the pa11y stuff:

1. Our current accessibility audit tests are broken - There's something wrong with puppetteer: https://github.com/pydata/pydata-sphinx-theme/runs/4822444059?check_suite_focus=true

   Run python docs/scripts/a11y.py --no-serve
   ##[debug]/usr/bin/bash -e /home/runner/work/_temp/7fe03fc5-38b4-4220-903d-0bf8c0a9da19.sh
   /home/runner/work/pydata-sphinx-theme/pydata-sphinx-theme/node_modules/puppeteer/lib/Page.js:215
   this.emit('error', new Error('Page crashed!'));
   

2. Our manual audit testing uses yarn but we no longer use it for the theme. I was investigating re-enabling the a11y.py check here, but realized that it is using yarn in that script, and we don't use yarn anymore in this package. So that script is broken now, I think.

   I do not understand the script or the yarn ecosystem well enough to debug it, just highlighting it here.

I think that the accessibility audits with pa11y was just flaky, not broken. I also opened up a new PR to reduce the scope to just lighthouse, and not touching pa11y. closing this one!